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MATERIALS AND METHODS FOR SIMULATING FOCAL SHIFTS IN 
VIEWERS USING LARGE DEPTH OF FOCUS DISPLAYS 
Related Applications 

This application is based on a co-pending provisional application, U.S. 
5 Serial No. 60/365,130, filed on March 15, 2002, the benefit of the filing date of 
which is hereby claimed under 35 U.S.C. § 1 19(e). 

Field of the Invention 
The present invention generally pertains to a method and a display for 
conveying depth in a real or virtual image, and more specifically, pertains to 
10 selectively applying blur cues to images portrayed on a large depth of focus 
(DOF) display, based on the accommodation (focus) or vergence of the viewer's 
eye, to increase the perception of depth by the viewer. 

Background of the Invention 
The world, as we experience it, is three-dimensional (3-D). However, 
15 although we experience a 3-D world, our senses do not directly receive 3-D data 
about the world. Instead, the optics of each eye project a two-dimensional (2-D) 
image onto the surface of the retina, and the visual system must infer data about 
the missing third dimension (i.e., depth) from those 2-D images and ft*om various 
supplementary depth cues. These depth cues include the oculomotor cues of 
20 vergence and accommodation^ and the stereoscopic cue of binocular disparity. 

Vergence: When a person's gaze is shifted to an object, the person'sicyes 
move to fixate on the object, that is, to place the retinal image of that object on the 
center of each eye's retina (the fovea), where the resolution of the eye is the 
highest. Oculomotor cues involve the sensing of the position of muscles in and 
25 around the eyes. One oculomotor cue, vergence^ refers to the phenomenon that 
lines of sight of the eyes are approximately parallel to one another when they are 



fixating on a very distant object, and the eyes rotate in toward each other (i.e., 
converge) as they fixate on closer objects. The brain receives sensory feedback 
regarding the relative eye positions, and this information serves as a depth cue. 

Accommodation: Like most cameras, the human eye has a limited DOF. 
5 When viewing a real scene, not every object in the scene is in focus at any given 
time. Instead, the viewer accommodates (adjusts the focus of the eye) to bring 
objects at various distances into focus. For instance, if the viewer accommodates 
to an object that is one meter away, the retinal image of an object that is 20 meters 
away is blurry. The farther away an object is from the focus point of the viewer, 

10 the blurrier the retinal image of that object is. 

The eye possesses a two part optical system. The cornea provides the 
majority of refraction (approximately 70 percent), but its refractive power is fixed. 
The crystalline lens sits behind the cornea, and its shape can be altered to increase 
or decrease its refractive power. 

15 When a fixated object is close to the observer, the ciliary muscles of the 

eye contract in order to make the crystalline lens more spherical and increase its 
refi:'active power, so that the image of the object is brought into focus on the 
retina. When a fixated object is far from the observer, the ciliary muscles of the 
eye relax, thereby flattening the lens and decreasing its refractive power 

20 (measured in diopters) so that the image of the object is brought into focus on the 
retina. Dioptric blur provides negative feedback that the accommodation control 
system uses when trying to accommodate to an object at a given distance at a 
given point in time. If a person looks at an object at a novel depth, it will be 
blurred from the initially inaccurate state of accommodation. If the system begins 

25 to shift accommodation in one direction, and the object becomes more blurry, this 
blur feedback causes the system to reverse the direction of the accommodation 
shift. If, instead, the. object becomes clearer, then accommodation continues to 
shift in the same direction. If the shift in accommodation overshoots the point of 
best focus, this manifests as increased blur, and the shift in accommodation 

30 reverses direction and slows. These shifts in accommodation continue until the 
blur feedback is minimized (the object comes into best focus). The process is 




dynamic, and the eye constantly monitors blur feedback and makes corrections in 
accommodation at rates up to 5 Hz. This process of natural viewing and focusing 
is known as closed-loop accommodation^ because the blur feedback loop is intact 
(or "closed"). The brain receives feedback about the state of activity of the ciliary 
5 muscles, providing the viewer with information about the depth of the object 
being viewed. 

Some viewing conditions artificially increase the DOF of the eye. For 
instance, if a scene is viewed through a small pinhole, then both distant and near 
objects are in focus at the same time. Under such conditions, the negative 

10 feedback of dioptric blur is removed or substantially decreased, and 
accommodation is said to be ''open-loop'" (because the feedback loop is 
interrupted). Under open-loop accommodation conditions, the viewer can 
accommodate from extremely near to far without a significant change in the 
retinal image of the scene. Some video displays can be made to have a very large 

15 DOF. As one example, the virtual retinal display (VRD) described in U.S. Patent 
No. 5,467,104 can have a large DOF, producing an open-loop accommodative 
response in users. Other large DOF displays can be fabricated and methods 
presented in this document are applicable to all possible large DOF displays. 

Vergence and Accommodation are Synkinetic: When one shifts one's 

20 gaze to an object at a given depth, the resultant vergence and accommodation 
responses are highly correlated. Not surprisingly, the accommodation and 
vergence mechanisms are synkinetic (an involuntary movement in accord with 
one mechanism is triggered when a movement in accord with the other 
mechanism occurs). This linkage can be observed in infants between three to six 

25 months old, suggesting a biological predisposition for the synkinesis. When the 
eye accommodates to a certain depth, the vergence system is automatically driven 
to converge to the same depth. Conversely, when the eye converges to a certain 
depth, the accommodation system is automatically driven to accommodate to the 
same depth. These cross couplings between accommodation and vergence are 

30 referred to as convergence driven accommodation and accommodation driven 
vergence. 



Binocular Disparity and Stereopsis: Another depth cue is binocular 
disparity. Because a small distance separates the two eyes, they have slightly 
different viewpoints, and hence different retinal images. In stereopsis, the visual 
system compares the images from the leflt and right eye, and makes inferences 
5 about the depth of objects based on disparities between the retinal locations on 
which the images of the objects fall. This depth cue has been exploited in 
stereograph ic displays (including Head Mounted Displays (HMDs) ), which 
present different images to each eye. 

An object at the point of fixation falls on corresponding points of the 

10 retina (the center of the fovea, in this case). Other objects at approximately the 
same depth as the fixated object will also fall on corresponding points of the 
retina. The imaginary curved plane that describes the area of space that will fall 
on corresponding retinal points is referred to as the horopter. Objects behind the 
horopter will be shifted toward the left side of the retina in the right eye and 

15 toward the right side of the retina in the left eye (i.e., the images are shifted 
toward the nose). Objects in front of the horopter will be shifted toward the right 
side of the retina in the right eye and toward the left side of the retina in the left 
eye (i.e., the images are shifted toward the ears). 

Interaction between Accommodation, Vergence, and Stereopsis: All of 

20 these depth cues interact. As mentioned previously, accommodation and vergence 
are synkinetic. Vergence and stereopsis interact. In order to stereoscopically flise 
objects at different distances, the eyes must converge to fixate upon those objects. 
The relative distance between right and left object images is greater for objects in 
the foreground of a stereographic image than for objects in the background of the 

25 image. As viewers use stereographic displays and look at objects in the 
foreground and background of the displayed scene, they must dynamically shift 
vergence. 

Accommodative Response to Current Non-Stereographic Video 
Displays: Research has indicated that viewers do not accurately focus their eyes 
30 on standard (non-stereographic) video displays (e.g., liquid crystal displays 
(LCDs) and cathode ray tubes (CRTs)). Their focus is, instead, biased in the 



direction of the resting point of accommodation (the degree of accommodation of 
the lens when a person is in a dark room or is otherwise deprived of an adequate 
visual stimulus). This resting point is not at the ciliary muscle relaxation point, 
which would produce a lens refractive power of 0 diopters, and varies between 
5 individuals. This inaccurate accommodation causes a video display to become 
somewhat blurred, and is thought to be a major contributor to the eye fatigue and 
headaches that often accompany prolonged video display use. It would thus be 
desirable to reduce these inaccuracies in accommodation and thereby reduce a 
cause of eye strain. 

10 Stereographic Video Displays: A number of video display manufacturers 

have attempted to increase the immersion and amount of information in the 
display by creating stereographic displays (the term stereoscopic display is often 
used interchangeably with this term). One example of a stereographic display is 
the stereoscopic head mounted display (HMD). Typically, HMDs consist of a 

15 helmet or set of goggles, with a separate small LCD screen set in front of each 
eye. Lenses are mounted between each LCD and eye and are typically configured 
to place the image at optical infinity. The images displayed on each LCD are not 
identical, but instead represent slightly different camera viewpoints. The left 
LCD displays the left half of a stereo image pair and the right LCD displays the 

20 right half. 

Another example of this display is the stereographic head tracked display 
(HTD). Two versions of HTD are common. With some HTDs, a user dons 
lightweight LCD shutter glasses with lenses that become opaque or transparent in 
synchrony with the frames displayed on a large table or wall mounted 2-D video 

25 display. When the shutter over the left eye is opened, the shutter over the right 
eye closes, and the left half of a stereoscopic image pair is flashed on the display. 
When the shutter over the left eye then closes, the shutter over the right eye opens, 
and the right half of the stereoscopic image pair is displayed. The opening of 
shutters alternates from side-to-side in quick succession, and the display 

30 synchronously shifts between showing left and right views, with every other 
frame displayed on the monitor reaching each eye. 



In other implementations of an HTD, the user wears glasses in which the 
left lens is polarized along an axis orthogonal to that of the right lens (e.g., the left 
lens is polarized vertically, while the right is polarized horizontally). The user 
looks at a screen, upon which the left stereo image has been projected with light 
5 polarized along one axis, and upon which the right image has been projected with 
light polarized along the other axis. Each lens admits only the light reflected from 
the screen that is of the matching polarization, so each eye is presented with a 
different image. Other implementations of stereographic displays include 
autostereoscopic displays, which enable users to view stereo images without 
10 wearing any special glasses. 

With all of these stereoscopic displays, the left eye receives only the left 
stereo image, while the right receives only the right stereo image, giving the user 
the illusion that he/she is looking into a virtual scene with two eyes at normal eye 
separation. 

15 Current Stereographic Displays Violate Accommodation-Vergence 

Synkinesis: Current stereographic displays, especially HMDs, tend to cause 
profound eye fatigue, often accompanied by headache. Research suggests that 
this eye fatigue is, in part, the result of an incompatibility between the 
construction of the display and the biology of accommodation. Specifically, the 

20 displays elicit a mismatch between the depth information provided by 
accommodation and that provided by vergence. The displays have a fixed plane 
of focus (usually at optical infinity, or 0 diopters) making it necessary for the 
viewer's eyes to maintain a static level of accommodation while using the display. 
If the viewer shifts his/her level of accommodation, the display becomes out of 

25 focus. Nonetheless, stereo displays also require that the viewer dynamically 
change the vergence of his/her eyes to binocularly fuse objects at different 
apparent distances. Accommodation and vergence are yoked — ^that is, they are 
synkinetically linked, such that when a person's eyes converge, the eyes also 
tends to accommodate near, and when a person's eyes decon verge, they tends to 

30 accommodate far. These displays violate this linkage and require a viewer to 
simultaneously converge his/her eyes while maintaining a fixed level of 
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accommodation. Accordingly, it would be desirable to provide a display that 
allows accommodation and vergence to shift synchronously, just as they do in 
natural viewing, and thus removes the primary source of stereo display eye strain 
and loss of visual acuity. 



The present invention employs a display method that, based on a measured 
or estimated accommodation (focus) of the viewer, selectively applies blur cues to 
images portrayed on a large DOF display. When using such a display in accord 
with the present invention, a viewer can look at a displayed scene (real or virtual) 

10 and naturally shift focus between objects in the scene. By accommodating to 
different objects, which may be at different distances from a real or virtual 
viewpoint, those objects come into sharp focus. The method gives the viewer the 
illusion of viewing a scene with objects at different depths in the scene, and in 
doing so, simulates natural viewing conditions. This method dramatically 

15 increases the interactivity of the display, which enhances the perceived realism of 
the display. The ability to accommodate to objects portrayed on a display can 
increase the impression of realism and the feeling of immersion in a displayed 
scene, reduce eye strain (because the viewer does not maintain static 
accommodation), and reduce computation time, since a computing device coupled 

20 to drive the display needs to devote fewer resources to render an object that is out 
of focus. When applied to stereographic displays, the system allows 
accommodation to move in synchrony with ocular vergence, reducing eye strain 
and the loss of visual acuity that results from a mismatch of accommodation and 
vergence angle. 

25 Brief Description of the Drawing Figures 

The foregoing aspects and many of the attendant advantages of this invention 
will become more readily appreciated as the same becomes better understood by 
reference to the following detailed description, when taken in conjunction with the 
accompanying drawings, wherein: 
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Summary of the Invention 




FIGURE 1 is a schematic diagram that illustrates how changes in the 
shape of a crystalline lens do not affect the focus of VRD images, due to the large 
DOF created by narrow beams of light forming a narrow exit pupil in an eye; 

FIGURE 2 is a schematic view illustrating one method in accord with the 
5 present invention, for creating a large DOF display; 

FIGURE 3 is a schematic diagram showing one embodiment for 
configuring a large DOF display and an accommodation monitoring device; 

FIGURE 4 is a schematic diagram showing another embodiment for 
configuring a large DOF display, with the accommodation monitoring device 
10 disposed perpendicular to the axis of the eye; 

FIGURE 5 is a schematic diagram showing yet another embodiment, with 
both the large DOF display and accommodation monitoring device disposed 
perpendicular to the axis of the eye, allowing the viewer a line of sight to the real 
world for augmented reality applications (virtual data superimposed over real 
15 world data); 

FIGURE 6 illustrates an embodiment in which a device to monitor 
accommodation is disposed off-axis, and shows that an optional binocular 
monitoring of vergence (relative to both eyes) can be made with a suitable device, 
to determine accommodation, while the viewer has a line of sight to the real world 
20 for augmented reality applications; 

FIGURE 7 is another embodiment in which an off-axis monitoring of 
accommodation is made; 

FIGURE 8 is a flow chart showing the logic employed in the present 
invention to select and display an image with an apparent focus plane that 
25 corresponds to a viewer's level of accommodation; 

FIGURE 9 illustrates an array of 2-D images that are pre-prepared for use 
in the present invention; and 

FIGURE 10 is a schematic diagram illustrating an arrangement for real- 
time focus control of a remote video camera with accommodation, wherein a 
30 viewer sees a real scene of a large DOF display, and accommodation is monitored 
dynamically. 




Description of the Preferred Embodiment 

Construction : 

The following components are needed to implement the present invention: 
(1) a display with a large DOF; (2) means for the (direct or indirect) measurement 
5 of accommodation; and, (3) a computing device that, at a minimum, receives the 
accommodation measurements, and can affect the input to the large DOF display. 
Each of these components is discussed in further detail below. 
Large DOF Displays : 

Any display with a large DOF can be used in this method. Several 

10 displays that fit this description are summarized here, but this does not exclude 
other possible displays with a large DOF. The provided examples of large DOF 
displays are not comprehensive, but rather serve as examples to illustrate the 
general principle of a large DOF display. Other large DOF displays can be used 
in place of the examples discussed below. 

15 VRD: The VRD is one type of display that can be used as a large DOF 

display in the present invention. As shown in FIGURE 1, the VRD used 
preferably comprises a scanned light display 21 (also known as a retinal scanning 
display, a scanned laser display, a scanning light display, a scanning laser display, 
a retinal scanned light display, a retinal light scanning display, a flying spot 

20 display). Scanned light display 21 includes a modulated light source 22 and an X- 
Y scanner 24. Light from the scanned light display is directed toward a concave 
mirror 26 through a beam splitter 28. The beam splitter 28 directs light reflected 
from concave mirror 26 toward a viewer's eye 30. Because all of the rays 
projected from the VRD converge at a very small exit pupil 36, the images it 

25 projects have a large DOF. The display creates the equivalent of a "pinhole" 
aperture at its exit pupil, enabling it to project a clear image on the back of a 
viewer's retina 34 independent of the degree of accommodation of a viewer's 
lens 32. In a view 40, a viewer's lens 32' is flatter, showing an accommodation to 
view an image that is farther away compared to lens 32 in a view 20, where 

30 lens 32 appears much rounder, as would be the case in viewing an image that is 
closer. However, because the image viewed by eye 30 is provided with a large 



DOF, the light rays that form the image are in focus on retina 34 in both view 20 
and view 40. Because all planes remain in similar focus throughout the range of 
accommodation of the viewer's eye, a viewer is provided with little or no 
feedback to the accommodative system, giving rise to open-loop accommodation. 
5 Small Aperture Viewing: The DOF of any display can be increased by 

viewing that display through a small aperture (such as a "pinhole" in a sheet of 
foil). Apertures of 0.5 mm diameter, or smaller, work best to generate open-loop 
accommodation while viewing a display, but apertures between 0.5 mm and 2 mm 
in diameter will partially open the loop, producing a satisfactory result. Viewing 

10 a display through a small aperture reduces the total amount of light entering the 
eye, so it is preferable to use fairly bright displays. If the device used to monitor 
accommodation uses non-visible wavelengths of electromagnetic radiation, the 
small aperture can be in a filter that is opaque to visible light, but transmits 
infrared light or some other non-visible wavelengths of electromagnetic radiation. 

15 Alternate Method for Creating a Large DOF Display: Very bright 

displays (including, but not limited to LCD projectors) can be modified to become 
large DOF displays. As shown in FIGURE 2, a pinhole aperture 52 can be placed 
at some point in front of an image source 54 (e.g., an LCD). Depending on the 
variety of image source or video projector actually used, the optics of the 

20 projector may be in front of or behind the small aperture, or may be removed 
completely. The beams emerging from the small aperture can be collected with 
optics 50 (which may comprise one or more lenses) that redirect the beams to 
form a small exit pupil 36 (an image of pinhole aperture 52). When the pupil of a 
viewer's eye 30 is placed to coincide with exit pupil 36, the viewer sees a 

25 Maxwellian view of image 37 on retina 34 of the projected image, with a large 
DOF. 

To adjust the divergence angle of the beams passing through the small 
aperture, an additional set of optics 56 may optionally be placed in front of the 
aperture. FIGURE 2 also shows the beam divergence (using the dotted lines on 
30 each side of one of the beams shown with dash lines). By comparing the beam 
characteristics in FIGURE 2 with those of the VRD in FIGURE 1, it can be seen 



that this display can be analogous to a parallel scan version of a scanned light 
display. That is, whereas the VRD scans a narrow beam serially, this alternative 
display projects all of the beams in parallel. It should be understood that the 
method (and variations thereof) described in this document for one of these 
S displays is also applicable to the other. 
Means for Monitoring Accommodation : 

Concurrent with the viewing of the large DOF display, the accommodation 
of the viewer must be monitored, to implement the present invention. A number 
of techniques can be used to monitor accommodation directly or indirectly. Some 

10 applicable devices include, but are not limited to, IR optometers. These devices 
objectively determine the degree of accommodation by analyzing the reflection of 
the IR light that they shine into a person's eye. Accommodation monitoring 
devices vary in the frequency with which they can make measurements. The 
more frequently they can make measurements, the better suited they are for use in 

1 5 the present invention. 

Direct Measurements of Accommodation: One example of an IR 
optometer that can be used in'this approach is the Welch Allyn, Model 14010 
SureSight™ autorefractor. Ho;wever, any similar device that provides information 
about a viewer's level of accommodation can be used in the place of the 

20 SureSight™ autorefractor. The SureSight™ autorefractor continuously measures 
the refractive power of the eye at 5 Hz. The measurement range for spherical 
refractive power is from +6.0 diopters to -5.0 diopters, and an additional 
measurement of cylindrical refractive power from 0 to +3.0 diopters along the 
axis of the cylinder is provided. One index of the level of accommodation of the 

25 viewer is the spherical refractive power; however, the spherical equivalent 
(spherical refractive power added to half of the cylindrical refractive power) 
generally provides a better index. The SureSight™ autorefractor has a working 
distance of 14 inches. 

The SureSight™ autorefractor is most accurate when its IR beam can enter 

30 into the eye along the eye's visual axis. A number of alternate configurations 
allow this to occur, and a subset of these configurations is illustrated in the 



embodiments of the present invention shown in FIGURES 3-7. In each of these 
Figures, a general layout of components is shown for a binocular display. These 
Figures are also helpful in visualizing the layouts for monocular displays by 
referring only to the right side of each Figure. One configuration that enables 
5 accommodation to be monitored while a viewer uses a large DOF display requires 
placing accommodation monitoring device 68 (such as the SureSight™ 
autorefractor or an equivalent) directly in front of the viewer's eye to directly 
transmit light from the eye through an optional lens 70. The image from the large 
DOF display is reflected into the eye with a beam splitter 66 (or cold mirror), as 

10 shown in FIGURE 3. The accommodation monitoring device takes 
accommodation measurements of an eye 62 (the viewers right eye in this 
example, but optionally a left eye 60 can be used), while the eye views the 
reflection of a large DOF display 54' in the Maxwellian view. The light beams 
from display 54' forms a small exit pupil or aperture that is aligned with the 

15 entrance pupil of the eye. Although two accommodation monitoring devices 
could optionally be used, it is not necessary to do so, because the accommodation 
levels of the right and left eyes are linked. Thus, for the viewer's left eye 60, only 
a beam splitter 64 (or mirror) is used to reflect large DOF display 54 into the left 
eye. 

20 Alternatively, accommodation monitoring device 68 is placed 

perpendicular to the line of sight of the viewer, as shown in the embodiment of 
FIGURE 4. A beam splitter or hot mirror 66', which in this embodiment 
preferably reflects only non-visible wavelengths of light but permits visible 
wavelengths to pass unimpeded, is mounted at a 45 degree angle, at the 

25 intersection between a viewer's line of sight and a measurement axis of the 
accommodation monitoring device. This configuration enables the viewer to look 
directly through the beam splitter and into the large DOF display, which is 
mounted along the viewer's line of sight, while the accommodation monitoring 
device continuously measures the level of accommodation via the reflection from 

30 the beam splitter. 



In another embodiment, which is shown in FIGURE 5, both 
accommodation monitoring device 68 and large DOF display 54* are disposed 
perpendicular to the line of sight of the viewer's eye 62. Beam splitter 66' reflects 
the non-visible measurement signal from accommodation monitoring device 68 
5 into the eye and also reflects the measurement signal from the eye back into 
lens 70 of the accommodation monitoring device. Beam splitter 64 (or a cold 
mirror), which transmits the non-visible measurement beam, reflects light from 
large DOF display 54' into eye 62, enabling the viewer to superimpose the 
displayed images with real objects 75. 

10 Other accommodation monitoring devices are accurate even when used 

off-axis. Accommodation monitoring device 69, which is of this type, can be 
mounted at an angle from the line of sight of the eye and monitor accommodation 
while the viewer views large DOF display 54' via a reflection in beam splitter 64 
(or a mirror) or directly, as shown respectively, in FIGURES 6 and 7. In FIGURE 

15 6, as in FIGURE 5, the viewer can optionally view real objects 75 through the 
display apparatus, while viewing superimposed displayed images. 

Indirect Estimates of Accommodation: Because vergence and 
accommodation are synkinetically linked, an alternative to measuring 
accommodation directly is to monitor the vergence angle of the eyes, and thereby 

20 indirectly estimate the accommodation of the viewer. Because changes in 
accommodation and vergence are highly correlated, a direct measure of vergence 
provides a satisfactory estimate of accommodation. Many devices are available 
that can quickly track eye movements (and thereby calculate yergence angle), and 
such devices are a relatively cheap alternative to devices that measure 

25 accommodation directly. In any of FIGURES 3-7, a vergence monitoring device 
can be substituted for the accommodation monitoring devices 68 or 69. 
Computing Device : 

A computing device 132 (shown in FIGURE 10) with custom software 
comprising a plurality of machine instructions that are stored and loaded in 

30 memory for execution by a processor in the computing device, or a device with a 
custom hardware board (not shown) that implements these steps, receives 



measurements from the accommodation monitoring device and converts those 
measurements into measurements of accommodative power. The algorithm 
accesses a database, identifies the image that best corresponds to a measured 
accommodative power, and outputs that image to the large DOF display for 
5 immediate display to the viewer. This process is repeated each time the 
accommodation monitoring device collects a new accommodation measurement 
(e.g., five times per second using the SureSight™ autorefractor for making 
accommodation measurements). 
General Description of Method 

10 The present invention uses simulated dioptric blur cues to "close" open- 

■ loop accommodation to a large DOF display. That is, a viewer using our system 
can look at a displayed scene (real or virtual) and naturally shift focus between the 
objects in that scene as if he or she were actually there. This effect is 
accomplished by the following means: 

15 The method employs a repeating cycle of steps, which are illustrated in 

FIGURE 8. The cycle may begin with any of the steps, and within one full 
iteration of the cycle, the blur information will be brought into accordance with 
the level of accommodation. Each iteration of the cycle updates the blur 
information and thus, enables the blur content of images to dynamically change in 

20 accordance with the viewer's accommodation. 

While a person views the large DOF display, the accommodation level of 
the viewer is continuously monitored with an external instrument, such instrument 
as the IR autorefractor, which was discussed above. However, as noted above, 
any accommodation monitoring device that provides information about the level 

25 of accommodation can be used in the place of the autorefractor. 

The accommodation measurements provide input for the computing device 
that selectively blurs objects in the scene, based on the current or anticipated level 
of the viewer's accommodation. For example, if the viewer accommodates to a 
distance of five meters, the computing device leaves objects near the five meters 

30 point in the scene unblurred and blurs other objects in proportion to their distance 



from the five meters focus point. The computing device outputs the selectively 
blurred image to the large DOF display. 

The viewer sees an updated view of the scene with appropriate blur 
information. Although the displayed image remains sharply optically focused on 
5 the viewer's retina at all times, the viewer perceives portions of the image to be 
blurry as if they were optically defocused. As the viewer shifts accommodation, 
the display dynamically adjusts the blur cues (bringing some objects into focus, 
and others out of focus) to match the accommodation level. 

The logical steps used in this process are shown in FIGURE 8. In a 
10 step 80, the person views the display and the person's eyes accommodate and 
converge on an element in the large DOF display. The accommodation of the 
viewer's eye(s) is measured in a step 82. Alternatively, the system measures the 
vergence (or gaze direction) of the viewer's eyes in a step 82' (and can then 
optionally compute the accommodation power based on the vergence). In a 
15 step 84, the measurement made in step 82 or 82' is adjusted by applying a 
predetermined calibration function, as appropriate. The calibrated result from 
step 84 is then used to quantify the level of accommodation exhibited by the 
viewer's eye(s) in a step 86. 

A decision step 88 determines if the image focus plane provided by the 
20 large DOF display matches the viewer's accommodation. If so, the logic proceeds 
with a step 90 that determines the image provided by the large DOF display need 
not be updated. Accordingly, in a step 92, the large DOF display continues to 
show the image (without any change in the image focus plane). However, if the 
results in decision step 88 indicate that the image focus plane does not match the 
25 viewer's accommodation as measured, the blur information in the image that is 
displayed must be altered. 

There are a number of alternative means by which the computing device 
can control the blur information in the image. Three alternate paths (A, B, and C) 
are illustrated in FIGURE 8, but other techniques can alternatively be used. 
30 Further details about these alternate methods employed by the computing device 
are discussed below. After one of these three approaches is used (i.e., after 



following the steps of path A, B, or C), the logic continues with a step 98 in which 
a new image is selected for display to the viewer, to provide the best focus plane 
corresponding to the current or anticipated level of accommodation by the viewer. 
A step 100 then updates the image to be displayed with the selected image, and 
5 step 92 displays the updated image on the large DOF display, as noted above. 
The logic then loops to step 80 to repeat the process. 
Source Images : 

A wide variety of images can be presented on the large DOF display. The 
source images of scenes can be obtained in a number of ways. A useful 

10 distinction can be drawn between two categories of images, i.e., the images 
presented on the large DOF display can depict either virtual scenes, or real 
scenes. A second useful distinction can be drawn between 2-Z> images derived 
from 3-D scenes in real-time, and pre-prepared 2-D images. The four 
combinations of these two categories lend themselves to different varieties of 

15 implementation. Each of these implementation variants is discussed in greater 
detail below, and for each implementation, a path in FIGURE 8 (A, B, or C) 
sketches its logical implementation. 

Virtual Scenes - Real-time Rendering: In connection with the logical 
steps of path A in FIGURE 8, a virtual 3-D scene is created in or transferred into a 

20 3-D modeling program. Three-dimensional rendering software can be used to 
render 2-D images of the virtual scene and add variable amounts of blur to 
selected virtual objects (sometimes referred to as rendering with "lens focus 
effects" or "DOF effects"). The 3-D rendering software simulates the limited 
DOF of natural vision by adding blur to those virtual objects that lie outside of a 

25 chosen plane of best focus. The plane of focus can be changed arbitrarily by a 
user or a program. In the preferred implementation for virtual scenes, the scene is 
rendered in real-time while the viewer uses the large DOF display and has his/her 
accommodation dynamically measured. Every time a new accommodation 
measurement is taken, the plane of best focus in the 3-D renderer is dynamically 

30 determined and shifted to match the current accommodation of the viewer in a 
step 94 of FIGURE 8, path A. In a step 96, the 3-D rendering software is then 



used to render a scene or image with a new focus plane that matches that 
corresponding to the accommodation measured for the viewer. The logic then 
proceeds with step 98. 

Using this implementation, the viewer can move and look around a virtual 
5 scene (using a joystick, mouse, motion trackers, or other input or pointing 
devices) and naturally accommodate to different objects in the virtual scene. The 
location of the viewer in the scene, the gaze direction, and accommodation of the 
viewer all are used to dynamically control the rendering options for the scene. 

While high resolution realistic real-time rendering requires significant 

10 computational power, use of accommodation information in the present invention 
can substantially decrease the computational power that is required for rendering 
the images displayed. The accommodation information can be used as a level of 
detail (LOD) effect, reducing the demand on the processor. Other conventional 
LOD effects are based on the distance from the person's viewpoint to other 

15 objects in a virtual scene (distant objects are rendered at a lower resolution than 
close objects). In these implementations, the accommodation of the eye provides 
an additional cue for LOD effects. Only objects on the focal plane in the virtual 
scene need to be rendered at the maximum resolution and contrast. All other 
objects off the focal plane can be rendered at a lower resolution and contrast, 

20 since the objects' high resolution content is subsequently masked by blurring. 
The farther an object is from the focal plane, the lower the resolution and contrast 
that are required to render the object so that the object looks natural. This method 
of using a viewer's accommodation information as a LOD effect reduces the total 
processing power necessary to render realistic high resolution 3-D scenes. 

25 Stereographic images (one for each eye) can be generated by rendering the 

virtual scene from two viewpoints. All of the methods described in this 
provisional patent can be applied to stereographic images. In addition, for all pf 
the approaches described herein for practicing the present invention, cylindrical or 
spherical image formats (such as the Apple Corporation's Quicktime™ VR 

30 format), and 2!4-D image formats can be used in place of 2-D images. Cylindrical 
or spherical image formats render a panoramic view of a scene and enable a 



viewer to rotate his/her virtual view of a scene without requiring the system to 
render separate images for each view rotation. 

The present embodiment enables a viewer to dynamically rotate the view 
of virtual scene without the scene needing to be re-rendered. Under this 
5 embodiment, a new image only needs to be re-rendered when the accommodation 
level of the viewer changes or the viewer moves within the scene. Two and 
one-half dimensional image formats retain the z-axis (depth) information for each 
pixel of a rendered image. Many 3-D rendering software packages separate the 
rendering of the 3-D scene from the rendering of the focus effects. First, they 

10 render a 2I/2-D image from the 3-D scene, then they render a selectively blurred 2- 
D image from the 2V2-D image. Using this method, the 3-D renderer only needs 
to render a new 2/2-D image when the viewer moves within a scene or rotates the 
view. If the viewer changes accommodation, the 2!/2-D image remains the same, 
and only the focus effects applied to it are changed. In addition, a hybrid 2*/2-D 

15 cylindrical or spherical image format, in which a cylindrical or spherical image 
contains z-axis (depth) information, can also be used in place of 2-D images in the 
herein described methods. In this embodiment, the 3-D renderer need only render 
a new 2V2-D image when the viewer moves through the scene or views moving 
objects. The focus effect only needs to be re-applied to the lYz-D image when the 

20 viewer accommodates— if the viewer rotates the view of the scene, the focus effect 
need not be updated. These methods reduce the total computation necessary to 
render a realistic high resolution 3-D scene. 

Virtual Scenes — Pre-Rendering: A computationally efficient 
implementation, which is suitable for use in connection with path B in FIGURE 8, 

25 renders a series of images from the virtual scene offline, but allows real-time 
control of the playback of the pre-rendered 2-D images. Before the viewer uses 
the large DOF display, a virtual scene is rendered into a number of 2-D images. 
In the simplest implementation of this embodiment, the objects in the 3-D scene 
and the virtual camera remain stationary across the series of rendered images. For 

30 each 2-D image, however, the location of the plane of best focus for the rendering 
effect is different. From the static scene, a continuum of images is generated, 



with the focus at optical infinity on one end of the continuum, and with the focus 
at a near point (e.g. 1 cm) at the other end. A one-dimensional (1-D) array of 2-D 
images is created and saved for future presentation. When the viewer uses the 
large DOF display and accommodates, the computing device selects the image 
5 from the array that was pre-rendered with a focal plane that matches the 
accommodation of the viewer, as indicated in a step 102 in FIGURE 8, and sends 
it to the large DOF display for immediate presentation in a step 104. The logic 
then proceeds with step 98, as described above. With this embodiment, the 
viewer sees a still scene on the display and may freely look around and shift focus 

10 between the various objects in the scene. 

Another method to handle virtual scenes with pre-rendering is similar to 
that discussed above, with the difference that a multi-dimensional array of 2-D 
images of a virtual scene is pre-rendered. As a simple example, a 2-D array 110 is 
shown in FIGURE 9 (higher dimensional arrays are also useful, but are more 

15 difficult to depict). Along one dimension of the array, e.g., the horizontal axis, the 
position of the plane of best focus varies and in other dimensions, e.g., along the 
vertical axis, different aspects of the scene rendering vary. In the exemplary 2-D 
array of FIGURE 9, 2-D images are created so that along the first dimension, the 
best focus plane varies and along the second dimension, the forward and 

20 backward position of the virtual camera varies. A third dimension can be 
incorporated into the array to represent animation of (an) object(s) within the 
scene. Additional dimensions can be used to represent more complex object 
animation, camera motion (e.g., pitch, yaw, translation, etc.), or other dynamic 
properties of the scene. This implementation gives the viewer greater freedom to 

25 move around an animated scene and dynamically accommodate to different 
objects, while keeping the demand on the processor of the computing device 
relatively low. 

For all of the embodiments described in this document, cylindrical or 
spherical image formats (such as the Apple Corporation's Quicktime™ VR 
30 format) or 2/2-D image formats can be used in place of 2-D images. Cylindrical 
or spherical image formats render a panoramic view of a scene, and enable a 




viewer to rotate his/her virtual view of a scene, without requiring the system to 
render separate images for each view rotation. Two and one-half dimensional 
image formats retain the z-axis (depth) information for each pixel of a rendered 
image. In this case, the images are pre-rendered without focus effects and the 
5 focus effects are rendered in real-time from the z-axis information. Many 3-D 
rendering software packages separate the rendering of the 3-D scene from the 
rendering of the focus effects. First, these software programs render a 2i4-D 
image from the 3-D scene, then they render a selectively blurred 2-D image from 
the 2V2-D image. In addition, a hybrid 214-D cylindrical or spherical image 

10 format, in which a cylindrical or spherical image contains z-axis (depth) 
information, can also be used in place of 2-D images in the present invention. 
Each of these image format alternatives are compatible with the single and multi- 
dimensional array embodiments described in this document. As with the real-time 
rendering implementation, stereographic images can be generated by pre- 

15 rendering the virtual scene from two viewpoints. All of the embodiments 
described herein can be applied to stereographic images. To conserve storage 
space, the images can be compressed, and/or the arrays of images can be 
converted into a video format and compressed using motion compression 
algorithms (such as that employed in the Moving Picture Experts Group (MPEG) 

20 algorithm). 

Real Scenes ~ Pre-Captured Images: This embodiment is also relevant to 
the steps of path B in FIGURES. As shown in regard to a system 130 in 
FIGURE 10, one method of generating images of real scenes is to employ a 
camera 136 (video or still, digital or analog) to acquire a number of images from a 

25 real 3-D scene 140, while the focus of camera lens 138 is varied across the 
images, from nearest focus to optical infinity (so that objects on different planes 
of depth will come into focus and out of focus across the set of images). If an 
analog camera is used, the pictures are digitized and loaded into the storage of 
computing device 132. The images are placed in an ordered array from farthest 

30 focus to nearest focus, and each image is coded to identify its best focused 
fixation plane. A database is generated that links images in the array to estimated 



accommodation levels (e.g., an image taken with the camera focused at infinity is 
linked with an accommodation estimate of 0.0 diopters). 

The operations of the computing device for this implementation are similar 
to those for the pre-rendered virtual scenes, supra. All of the methods previously 
5 described for acquiring, representing, and handling images of virtual scenes are 
also applicable to the pre-captured images of real scenes. For instance, the 
captured images can be formed into multi-dimensional arrays to allow for greater 
interactivity with the displayed scene (e.g., the viewer and/or objects can move 
within a scene) and cylindrical and spherical image formats, 2V2-D image formats, 

10 and 2/4-D cylindrical and spherical image formats can be used in place of 
.standard 2-D images. 

Stereo images are generated by acquiring images from two viewpoints in a 
scene (two cameras can be used simultaneously, or one camera can be moved 
between to viewpoints serially). The viewpoints can be equal in separation to that 

15 of human eyes, or they can be farther apart to exaggerate the stereo information 
(or closer together to decrease the stereo disparity). In all of the methods 
mentioned herein, stereo images can be substituted for images. 

Multiple cameras with different focal lengths can be used to 
simultaneously acquire images from the same scene. A conventional software 

20 algorithm is used to interpolate intermediate images between the acquired images 
from the different cameras (e.g., if one camera is focused at 4 diopters and the 
other camera is focused at 5 diopters, the software algorithm is used to interpolate 
images with a focus of 4.5 diopters). This approach enables a set of images with 
varied focus planes to be generated for a moving scene. When performed with a 

25 set of still cameras, the method creates a set of pictures of the same freeze-frame 
of a moving scene. When performed with a set of video cameras, the method can 
create a set of movie channels that are identical except for their focus planes. 
When the movies are played back, the accommodation of the viewer controls the 
channel that is displayed on the large DOF display. Thus, the viewer is able to 

30 watch a movie, while dynamically accommodating to different objects in the 
movie scene. 




Real Scenes — Real-time Camera Control: This embodiment is relevant 
to the steps of path C in FIGURE 8. One or more video cameras (digital or 
analog) are used to film real scene 140, generally as shown in FIGURE 10. This 
footage is relayed to large DOF display 54' in real-time, so the viewer can see the 
5 scene from the camera's point of view. Video camera 136 can be in the vicinity 
of the viewer, to show the viewer's own environment, or the video camera can be 
at a remote location, enabling telepresence and teleoperation abilities. The focus 
of the camera is changed with an actuator 134. The computing device controls the 
focus by sending signals to the actuator, and the focus is dynamically changed (a 

10 step 108 in FIGURES) to match the accommodation level of the viewer, as 
indicated in a step 106 of FIGURE 8. Two cameras can be used to create stereo 
video images. The accommodation of the viewer measured with accommodation 
monitoring device 68 provides a signal to computing device 132, which controls 
the focus for each of the cameras in response, as shown in FIGURE 10. 

15 In addition to measuring accommodation, the motion of the viewer's head 

can be tracked, and the orientation of the camera(s) (e.g., pan, tilt, yaw, etc.) can 
be actuated to follow the head motion. Optionally, the vergence angle of the eyes 
can control the vergence angle of the cameras. The zoom of the lens can be 
linked to an input device (e.g., joystick or keyboard) to allow the viewer to move 

20 in and out of the scene. These features enable a viewer to look around a remote 
real scene in real-time, accommodate naturally to objects in the scene, and zoom 
in on objects. 

Another embodiment uses a similar arrangement, with the exception that 
the accommodation of the viewer controls another mechanism of the camera. For 

25 instance, the zoom level of the camera can be linked to the accommodation of the 
viewer. When the viewer accommodates to a distant object in the scene, the 
camera zooms in towards that object. In another embodiment of this concept, the 
accommodation of the viewer can be used to control a linear actuator that 
physically translates a camera forward and backward relative to the scene being 

30 recorded with the camera. 




Other Modifications : 

A number of exemplary implementations of the present invention have 
been discussed above. Combinations of any the elements from the exemplary 
implementations can be formed into many other implementation variants, which 
5 are too numerous to specifically describe. Other non-described implementations 
that combine elements of these described implementations are also claimed. 

All of the embodiments described above can be applied to stereographic 
images, such as varying the focus of a stereoscopic pair of cameras instead of only 
a single camera. In addition, for all of the embodiments described herein, 
10 cylindrical or spherical image formats (such as the Quicktime™ VR format) and 
21/2-D image formats can be used in place of 2-D images. 

Integrated Large DOF Display with Infrared (IR) Eye Tracking: The 
accommodation and/or vergence monitoring device can be incorporated into the 
large DOF display. They are listed as separate components in the rest of this 
1 5 disclosure, but they can be combined into one physical unit. This combination of 
components makes the system lighter, more compact, and is expected to be more 
energy efficient. 

For example using the VRD, constant intensity IR laser light can be added 
to the modulated visible light source of the VRD before the beams are scanned to 

20 the eye. The intensity and distribution of IR light beams reflected fi-om the eye 
will depend on the viewer's line of sight within the display. If the gaze of the eye 
is aligned with the collimated beam of IR light, then the angle of incidence of the 
incident beam will have zero degrees. After striking the apex of the viewer's 
cornea, the reflected beam will broaden while returning through the X-Y scanners 

25 to the original source. Typically, the source is an optical fiber. Therefore, 
surrounding the source fiber can be one or more rings of optical fibers to collect 
the reflected beam of light. Only when the IR beam of light is aligned with the 
viewer's line of sight will the reflected beam have a high intensity that is 
distributed symmetrically about the central axis, corresponding to the central 

30 source fiber. Thus, equal intensities of light should be collected in each optical 
fiber within each ring when the scanned beam of IR light is aligned with the 




viewer's line of sight. When the IR beam is not aligned with the viewer's line of 
sight, the reflected beam is not centered about the central source fiber among the 
ring of collection fibers. Accordingly, some collection fibers will have much 
lower intensities than collection fibers on the opposite side of the ring. During 
5 each frame of the display, the scanned IR beam and its distribution of reflected 
light will track which pixel the viewer is centrally fixated on. By determining the 
scan position or pixel being fixated on by both left and right eyes of the viewer, 
the vergence angle can be calculated and accommodation can be inferred. 

A large DOF display with an array of multiple exit pupils can be used to 

10 enable users to change eye position within a large field of view display without 
"losing" the single small exit pupil. To prevent multiple exit pupils from entering 
the viewer's eye, the array can be spaced apart such that only one exit pupil can 
enter the eye at any one time. Another option is to turn off all exit pupils not 
centrally aligned with the viewer's gaze. Crude eye-tracking can be used to 

15 determine the exit pupil that should be active. Alternatively, eye-tracking can be 
used to actively keep a single exit pupil centrally aligned with the viewer's eye. 

As people age, the crystalline lens becomes less elastic, and the maximum 
magnitude of their accommodative range decreases. This reduced ability to 
accommodate or vary focus is termed presbyopia. The amplitude of 

20 accommodation shifts of older viewers can be amplified, so that small changes in 
accommodation result in larger shifts of the simulated focal plane in the large 
DOF display. The total range of youthful accommodation can be mapped onto the 
restricted range of older viewers. For instance, if an older viewer can only 
accommodate to a maximum of 2 diopters, that accommodation level can be 

25 linked with images displaying a 10 diopter focus level. 

The accommodation of the viewer can be extended beyond the natural 
range of human eyes, such as looking and focusing very close to an object. The 
ability to maintain focus will produce a magnifying effect in a display, such as 
looking through a magnifying glass or microscope. 

30 Non-natural fixation depths can be used to enable a viewer to look into 

regions not normally accessible. For instance, a doctor can use a large DOF see- 




through HMD (like that of FIGURE 5) while examining a patient. Reconstructed 
data from imaging scans (e.g., fMRI, MRI, CAT, PET, EEG, MEG, EKG, x-ray, 
etc.) can be superimposed over the doctor's real view of the patient. The doctor 
can accommodate to the level of the patient's skin and see the real view of the 
5 skin. As the doctor shifts the accommodation and vergence level in towards the 
inside of patient's body, an x-ray view of the patient's broken bone can be 
brought into view. The interface can be configured such that the bone only comes 
into focus when the doctor accommodates to its real depth. This provides the 
doctor with an intuitive feel for the location of the bone break, and may help the 

10 doctor find an internal site more rapidly and efficiently when performing surgery, 
for example, using laparoscopic techniques. By applying the natural oculomotor 
processes of accommodation and vergence to non-natural data sets (like x-ray 
data), those data sets may be understood more easily. This approach can be used 
for general data visualization applications. The accommodation and vergence 

1 5 level provides a natural means of controlling an additional variable when sorting 
through any data set. 

Changes in Depth of Field: Using the methods described in this 
document, one can render images from a virtual scene with any desired depth of 
field. A DOF that approximates that of the normal human eye allows for the most 

20 natural perception of a scene. A DOF that is more narrow than that of normal 
human vision may exaggerate the role of focus on perception of the scene. A 
hyperreal focus effect may be achieved. The role of focus will be more salient, 
and this salience may help viewers with accommodative dysfunction, such as the 
inability to focus accurately, to improve their accuracy. By exaggerating the 

25 errors of incorrect accommodation levels, the feedback to the accommodation 
system is increased. This display system could be used as an accommodation 
trainer for those who suffer from accommodative dysfunction. 

Non-Display Applications: The focus level of the eye provides data to the 
computing device about which objects are being attended to. A disabled user 

30 could interact with real and virtual scenes by shifting focus to an object. A 3-D 




eye tracking system and 3-D mouse can be fomied with the accommodation 
measurement method. 

Simulated Longitudinal Chromatic Aberration: In addition to dioptric 
blur, the human accommodation system also receives feedback in the form of 
5 longitudinal chromatic aberration (LCA) when the eye accommodates. Short 
wavelengths (blue) are more strongly refracted than medium wavelengths (green), 
which in turn are more strongly refracted than long wavelengths (red). Thus for a 
given level of accommodation, the red, green and blue channels of an image are 
not equally blurry. The differences in blur across color channels provide the 

10 human accommodation system with additional feedback to control 
accommodation. All of the methods in this document that have been applied to 
blur information can be equally applied to LCA information. The LCA can be 
simulated for images in addition to (or as an alternative to) the simulation of blur. 
Computing device images, like the color photoreceptors of the eye, are divided 

1 5 into red, green, and blue components (or "channels"). In natural vision, the optics 
of the eye do not refract each wavelength of light to the same extent. 

Although the present invention has been described in connection with the 
preferred form of practicing it and modifications thereto, those of ordinary skill in the 
art vsall understand that many other modifications can be made to the present 

20 invention within the scope of the claims that follow. Accordingly, it is not intended 
that the scope of the invention in any way be limited by the above description, but 
instead be determined entirely by reference to the claims that follow. 



